Spatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition

Transformer-based methods have recently achieved great advancement on 2D image-based vision tasks. For 3D video-based tasks such as action recognition, however, directly applying spatiotemporal transformers video data will bring heavy computation and memory burdens due to the largely increased number of patches quadratic complexity self-attention computation. How efficiently effectively model has been a challenge for transformers. In this paper, we propose Temporal Patch Shift (TPS) method efficient modeling in recognition. TPS shifts part with specific mosaic pattern temporal dimension, thus converting vanilla spatial operation one little additional cost. As result, can compute using nearly same cost self-attention. is plug-and-play module be inserted into existing transformer models enhance feature learning. The proposed achieves competitive performance state-of-the-arts Something-something V1 & V2, Diving-48, Kinetics400 while being much more source code found at https://github.com/MartinXM/TPS.